Few-Shot Domain Adaptation with Polymorphic Transformers

Deep neural networks (DNNs) trained on one set of medical images often experience severe performance drop unseen test images, due to various domain discrepancy between the training (source domain) and (target domain), which raises a adaptation issue. In clinical settings, it is difficult collect enough annotated target data in short period. Few-shot adaptation, i.e., adapting model with handful annotations, highly practical useful this case. paper, we propose Polymorphic Transformer (Polyformer), can be incorporated into any DNN backbones for few-shot adaptation. Specifically, after polyformer layer inserted source domain, extracts prototype embeddings, viewed as “basis” source-domain features. On adapts by only updating projection controls interactions image features embeddings. All other weights (except BatchNorm parameters) are frozen during Thus, chance overfitting annotations greatly reduced, perform robustly being few images. We demonstrate effectiveness Polyformer two segmentation tasks (i.e., optic disc/cup segmentation, polyp segmentation). The code released at https://github.com/askerlee/segtran.